This invention relates to training firefighters in an augmented reality (AR) simulation that includes creation of graphics depicting fire, smoke, and application of an extinguishing agent; and displaying the simulated phenomena anchored to real-world locations seen through a head-worn display.
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office records but otherwise reserves all copyright works whatsoever.
Current fire simulation for firefighter training is accomplished at facilities that use propane burners and extinguishing agent collectors to simulate the behavior of various types of fires. This approach presents numerous disadvantages, such as safety risks attributable to unintended reflash and explosion; environmental damage attributable to combustion byproducts; health risks to crews due to inhalable combustion byproducts; high operation costs attributable to fuel requirements; high maintenance costs to ensure system integrity and safety; and unrealistic fire simulations for some types of fires (all simulations appear as propane fires as opposed to oil, electrical or paper; and simulated smoke is white instead of black).
A need exists for a new generation of fire fighting/damage control simulation system which does not use live fires. These systems must be capable of providing a high fidelity representation of the smoke and flames, as well as a realistic representation of the environment (to include fellow crew members). Augmented reality (AR) technology allows overlay of computer-generated graphics on a person""s view of the real world. With AR, computer generated fire, smoke, and extinguishing agents can safely replace live fire training while still allowing trainees to view and interact with each other and the real-world environment. This allows safe, cost-effective training with greater realism than pure virtual reality (VR) simulations.
The majority of current generation of fire fighting training systems use live, propane-based fires which are unsafe, particularly for use in contained areas such as onboard ships, and in real structures. In a training environment, the use of live propane-based fires presents safety, health and environmental risks.
The primary objective of this invention is the development of an augmented reality-based training (ARBT) system for fire fighting, with application to rescue and hazardous material mitigation. In fact, in any fire situation there are multiple goals, including:
Search, rescue, and extrication
Ingress into, and egress from, a structure
Fire suppression
Structure stabilization
Team coordinationxe2x80x94command and control
Fire cause determination
In each of the goals, firefighters engage in a number of cognitive and physical tasks critical to the survival of both fire victims and firefighters, as well as to the timely suppression of a fire. Tasks that fall under this category are
(1) Navigation
(2) Situation awareness
(3) Decision making/problem solving
(4) Stress management
These tasks are undertaken, usually in concert with one another, to achieve the above goals. Training in these four tasks provides the foundation for a firefighter to combat any fire situation. An opportunity exists to develop an ARBT system which educates firefighters in these tasks in a safe and potentially less expensive environment, in almost any location.
It is important at this juncture to distinguish between the concept of reaction versus interaction with fire and smoke. By reaction we connote responses made by a firefighter to conditions caused by fire and smoke; in this situation he/she does not alter the evolution of the fire and smoke. By interaction we mean that the firefighter directly affects the evolution of the fire and smoke by such actions as fire suppression and ventilation. As stated above, Tasks (1) to (4) are applicable to any fire situationxe2x80x94reactive or interactive. Therefore, any significant improvement in developing training skills for Tasks (1) to (4) will result in a significantly skilled firefighter for both reactive and interactive scenarios.
An objective of this invention is to demonstrate the feasibility of augmented reality as the basis for an untethered, ARBT system to train firefighters. Two enabling technologies will be exploited: a flexible, wearable belt PC and an augmented reality head-mounted display (HMD).
Unlike traditional augmented reality systems in which an individual is tied to a large workstation by cables from head mounted displays and position trackers, the computer technology is worn by an individual, resulting in an untethered, augmented reality system.
Augmented reality is a hybrid of a virtual world and the physical world in which virtual stimuli (e.g. visual, acoustic, thermal, olfactory) are dynamically superimposed on sensory stimuli from the physical world.
This invention demonstrates a foundation for developing a prototype untethered ARBT system which will support the critical fire fighting tasks of (1) navigation, (2) situation awareness, (3) stress management, and (4) problem solving. The system and method of this invention can be not only a low-cost training tool for fire academies and community fire departments, but also provides a test bed for evaluating future fire fighting technologies, such as decision aids, heads-up displays, and global positioning systems for the 21st century firefighter.
Accordingly, the primary opportunity for an ARBT system is the training of firefighters in the areas of Tasks (1) to (4) above for reactive scenarios.
Overall Payoffs. The inventive ARBT system has the significant potential to produce
Increased safety
Increased task performance
Decreased workload
Reduced operating costs
A training program that aims to increase skills in the Tasks (1) to (4) is adaptable to essentially any fire department, large or small, whether on land, air, or sea.
Opportunities for Augmented Reality for Training. Augmented reality has emerged as a training tool. Augmented reality can be a medium for successful delivery of training. The cost of an effective training program built around augmented reality-based systems arises primarily from considerations of the computational complexity and the number of senses required by the training exercises. Because of the value of training firefighters in Tasks (1) to (4) for any fire situation, and because the program emphasizes firefighter reactions to (vs. interactions with) fire and smoke, training scenarios can be precomputed.
As described elsewhere in this document, models exist which can predict the evolution of fire and smoke suitable for training applications. An opportunity exists to exercise these models off line to compute reactive fire fighting scenarios. These precomputations can lay out various fire-and-smoke induced phenomena which evolve dynamically in time and space and can produce multi-sensor stimuli to the firefighter in 3D space. (For example, if the firefighter stands up, he/she may find his/her visibility reduced due to smoke, whereas if he/she crawls, he/she can see more clearly.)
It has been demonstrated that PC technology is capable of generating virtual world stimulixe2x80x94in real time. We can then apply our augmented reality capabilities to the development of an augmented reality-based training system.
In summary, the opportunity identified abovexe2x80x94which has focused on reactions of firefighters to fire and smoke in training scenariosxe2x80x94is amenable to augmented reality.
Opportunities for Augmented Reality for Training. In augmented reality, sensory stimuli from portions of a virtual world are superimposed on sensory stimuli from the real world. If we consider a continuous scale going from the physical world to completely virtual worlds, then hybrid situations are termed augmented reality.
The position on a reality scale is determined by the ratio of virtual world sensory information to real world information. This invention creates a firefighter training solution that builds on the concept of an augmented physical world, known as augmented reality. Ideally, all training should take place in the real world. However, due to such factors as cost, safety, and environment, we have moved some or all of the hazards of the real world to the virtual world while maintaining the critical training parameters of the real world, e.g., we are superimposing virtual fire and smoke onto the real world.
For a fire example, consider the following. Suppose an office room fire were to be addressed using augmented reality. In this problem, a real room with real furniture is visible in real time through a head mounted display (HMD) with position tracker. Virtual fire and smoke due to virtual combustion of office furniture can be superimposed on the HMD view of the physical office without ever having to actually ignite a piece of real furniture.
The inventive approach allows the firefighter to both react and interact with the real world components and the virtual components of the augmented reality. Examples of potential real-world experiences to be offered by our approach are given below in Table 1xe2x80x941.
Clearly, simulation of training problems for firefighters can comprise both physical and virtual elements. In many instances augmented reality may be a superior approach when compared to completely virtual reality. For example, exercise simulators such as stationary bicycles, treadmills or stair climbing machines do not adequately capture either the physical perception or the distribution of workload on the musculoskeletal systems that would be produced by actually walking or crawling in the physical world. Additionally, a firefighter can see his/her fellow firefighters, not just a computer representation as in pure virtual reality.
Opportunities for Self-Contained Augmented Reality. A low-cost, flexible, wearable belt PC technology may be used in augmented reality firefighter training. This technology, combined with augmented reality and precomputed fire scenarios to handle tasks (1) to (4) above for various physical locations, allows a firefighter to move untethered anywhere, anytime, inexpensively and safely. This will significantly add more realistic training experiences.
Background Review of Fire Simulation. Mitler (1991) divides fire models into two basic categories: deterministic and stochastic models. Deterministic models are further divided into zone models, field models, hybrid zone/field models, and network models. For purposes of practicality and space limitations, we limit the following discussions to deterministic models, specifically zone type fire models. Mitler goes on to prescribe that any good fire model must describe convective heat and mass transfer, radiative heat transfer, ignition, pyrolysis and the formation of soot. For our purposes, models of flame structure are also of importance.
Zone models are based on finite element analysis (FEA). In a zone model of a fire, a region is divided into a few control volumesxe2x80x94zones. The conditions within each volume are usually assumed to be approximately constant. In the study of compartment fires, two or more zones typically are used: an upper layer, a lower layer, and, optionally, the fire plume, the ceiling, and, if present, a vent. Zone models take the form of an initial value problem for a system of differential and algebraic equations. Limitations of zone models include ambiguity in the number and location of zones, doubt on the validity of empirical expressions used to describe processes within and between zones, and inapplicability of zones to structures with large area or complex internal configurations.
For many training applications, such effects are not significant for purposes of this invention. Friedman (1992) performed a survey of fire and smoke models. Of 31 models of compartment fire, Friedman found 21 zone models and 10 field models. Most of the zone models can run on a PC, while most of the field models require more powerful computational resources.
Background Review of Virtual Reality-Based Trainingxe2x80x94and Potential for Augmented Reality-Based Training. Probably the core issue surrounding the development of any training system or program is the efficiency of the transfer of knowledge and skills back into the workplace. Individual development ultimately rests on the ability to adapt acquired skills to novel situations. This is referred to, by some, as a metaskill. The transference of skills and the building of metaskills are fundamental concepts against which virtual reality must be considered for its suitability as a basis for the delivery of training.
Experiential learning is based on the premise that people best learn new skills by successfully performing tasks requiring those skills. The application of virtual reality to the delivery of training builds on the promise of experiential learning to maximize the transfer of training into the task environment. Furthermore, virtual reality interfaces also hold the potential for being more motivating than traditional training delivery media by making the training experience itself more fun and interesting. Augmented reality retains these strengths while providing a real world experience for the firefighter.
When concerned with the transfer of skills from a virtual world to the real world, the issue of virtual world fidelity is often raised. Alessi (1988) examined the issue of simulator fidelity in both initial learning and transfer of learning and found that the impact of simulator fidelity increases with the level of expertise of the student. He goes on to recommend that fidelity increase along lines of instruction phases: presentation, guidance, practice, and assessment. Alessi""s results are corroborated by Lintern et al. (1990) in their work on the transfer of training using flight simulators for initial training of landing skills. Most notably the authors found that feedback, related to correct performance of the landing task, resulted in increased transfer of training. They also found that transfer of training did not necessarily increase with increasing simulator fidelity. These results on fidelity are important in that they emphasize that simply creating a task environment in a virtual world without consideration of learning processes may not be sufficient to transfer skills to the physical world.
Review of Augmented Reality Equipment. A description of augmented reality was presented above. Commercial off the shelf technologies exist with which to implement augmented reality applications. This includes helmet-mounted displays (HMDs), position tracking equipment, and live/virtual mixing of imagery.